Timestamp not being respected in eventv2 PD-CEF (using Go package)

peter.galbavy · October 20, 2022, 7:27am

I am working on renewing our event integration to PD and I am using the Go package (github.com/PagerDuty/go-pagerduty).

The only part that doesn’t seem to work is setting the timestamp in the payload to the actual time of the event. I have tried various ISO8601 / RFC3339 formats, just in case it is fussy, but none seem to work.

Am I pushing water uphill or should this work?

The formats I have tried are time.RFC3339 and then changing the timezone to match the examples in the dev docs (i.e. 2006-01-02T15:04:05-0700 and 2006-01-02T15:04:05-07:00). I can see the correct test value is passed to ManageEventWithContext() but in the dev web interface all events raise incidents with wall time and not the embedded value (from yesterday or an hour ago, in case there is a limit).

When viewing the underlying message in the web side it is in the old V1 format not PD-CEF and so carries no time information.

Any ideas or guidance, please?

Thanks!

demitri · October 24, 2022, 10:56pm

What is the type of the integration corresponding to the key in use? If all events were showing up as V1 when received then I would first check to make sure that it is an Events API V2 integration. If that’s not already the case then it would be, as you say, like pushing water uphill. Otherwise, it could possibly be something to look into with respect to the client or how it is configured.

peter.galbavy · October 25, 2022, 7:20am

I am developing the next version of our integration, so I am using Custom Event Transformer as this appears to be the only option open to me.

Yes, I was also confused about v1 / v2 - but I am using the Go package (very simplisticly):

_, err = client.ManageEventWithContext(context.Background(), &v2event)

Code, if you need to look, is in dev here:

Thanks!

dmcclure · October 25, 2022, 12:56pm

I’m pretty certain that it’s not possible to set a timestamp in the event payload. This is set automatically when ingested into the event API. If you need to include some other kind of timestamp reference, add that as a new field under custom_details.

demitri · October 25, 2022, 3:59pm

I would recommend trying with a plain v2 events integration. The go-pagerduty client already emits an event that is structured appropriately for that kind of integration as far as I can tell, so that should save you a lot of effort.

A Custom Event Transformer integration would be more useful for a case where the service sending the event cannot be customized to make the event payload an appropriate structure for a plain PagerDuty integration. It allows the PagerDuty side to accommodate the special inflexible structures.

However, seeing as you’re using a mainstream PagerDuty API client, that shouldn’t be necessary; it already should format the event for the standard v2 Events integration.

If you’re developing an integration for public use and need some kind of branding in the PagerDuty UI when selecting it, then there might be a process to do that and have it be basically a clone of the standard V2 Events integration (I don’t know off the top of my head, but we can ask).

peter.galbavy · October 25, 2022, 4:39pm

Thanks for the suggestions. I may simply be confused, as in reading the docs I was under the impression that the Custom Event Transformer is the “fallback” integration that you have to use, as the API requires a routing key, which is the integration key from the specific service.

I did read https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw-events-api-v2-overview and I do see:

but I cannot see how/where that happens.

When searching my dev instance I see this even searching

The other entries are branded integrations with the word “event” in the name.

Have I misunderstood?

Sorry, but Pagerduty is not my primary expertise by a long long way.

peter.galbavy · October 25, 2022, 4:45pm

According to the docs - if I could only understand the link to the v2 API - says you can:

https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw-events-api-v2-overview#pagerduty-common-event-format-pd-cef

peter.galbavy · October 25, 2022, 4:48pm

Hang on, it’s there in the icon list, just doesn’t appear when typing it in! Let me try that now

.

demitri · October 25, 2022, 4:49pm

I similarly tried going through the add integration UI to confirm and I saw Events API V2 showing up under “Popular Integrations”, but it does not show up in results when I search for it by name.

peter.galbavy · October 25, 2022, 4:57pm

Right, I have now removed the CET and added the Event V2 integration, updated the routing key in my config and tried again but the “message” still looks v1 and contains no timestamp.

Sorry for the noddy issues - but I am hitting brick walls here

demitri · October 25, 2022, 5:24pm

Do you mind sharing a code snippet where the V2Event typed object is constructed?

The property that needs to be set is V2Event -> V2Payload -> Timestamp

(source code)

peter.galbavy · October 25, 2022, 5:30pm

Sure, it’s OSS:

The timestamp is set here (we export ANSIC, so parsing is required) - https://github.com/ITRS-Group/cordial/blob/04e93db2d4cc93b9ac75d08fe18df3a07bccbf46/integrations/pagerduty/cmd/root.go#L132

	timestamp := payload.GetString("timestamp")
	if timestamp == "" {
		timestamp = time.Now().Format(time.RFC3339)
	} else {
		// geneos timestamp format is Go ANSIC format
		t, err := time.Parse(time.ANSIC, timestamp)
		if err != nil {
			timestamp = time.Now().Format(time.RFC3339)
		} else {
			timestamp = t.Format(time.RFC3339)
		}
	}

The struct is populated here: https://github.com/ITRS-Group/cordial/blob/04e93db2d4cc93b9ac75d08fe18df3a07bccbf46/integrations/pagerduty/cmd/root.go#L177

	v2event := pagerduty.V2Event{
		RoutingKey: routing_key,
		Payload: &pagerduty.V2Payload{
			Summary:   payload.GetString("summary"),
			Source:    payload.GetString("source"),
			Severity:  severity,
			Timestamp: timestamp,
			Group:     payload.GetString("group"),
			Class:     payload.GetString("class"),
			Details:   details,
		},
		DedupKey:  cf.GetString("pagerduty.event.dedup-key"),
		Client:    cf.GetString("pagerduty.event.client"),
		ClientURL: cf.GetString("pagerduty.event.client_url"),
		Action:    action,
		Links:     links,
		Images:    images,
	}

	_, err = client.ManageEventWithContext(context.Background(), &v2event)

demitri · October 25, 2022, 6:28pm

That being the case, I’m a bit stumped myself.

Anyone with more extensive Go experience might be able to help?

If the body of the resultant HTTP request (the next thing I’d check, but I don’t know how) includes that field and the value supplied, then the next step I’d recommend is opening a support ticket to examine how the events are being processed, and escalate the issue if necessary.

peter.galbavy · October 25, 2022, 7:09pm

I have not yet tried posting raw JSON to test outside the Go paxkage. That can be next and then if that fails I will try support.

Thanks.

smcallister · October 25, 2022, 11:18pm

Hi Peter! You won’t see the payload timestamp value in the “message”. As Doug mentioned above, you currently can’t set the timestamp through the API. If you want to include a time stamp value you can set a custom field value on the event payload.

It appears from your code that you’re already setting a value for Payload.Details, so you can add a timestamp field to your existing Details struct. As a reference for those who may have a similar issue in the future and find this post, setting the Details field would look something like this:

type Details struct {
	Timestamp string `json:"timestamp"`
}

dets := Details{"2022-10-20T08:42:58.315+0000"}

payload := pagerduty.V2Payload{
	Summary:  "Get the door!",
	Severity: "critical",
	Source:   "go-library",
	Details:  dets,
}

Then, the timestamp value would be visible in the Custom Details section of the alert, as seen here:

And, in the Event Details message, like this:

peter.galbavy · October 26, 2022, 11:06am

Thanks, I was working on the assumption it did - according to the PD-CEF docs anyway. Obviously, the Event v2 API is not the same thing but nothing gave me the hint it would not work.

I will follow your suggestion (or rather document it for our config file) and leave it at that for now.

peter.galbavy · October 31, 2022, 10:51am

To follow this up and in case others get to this thread, it seems if I use an Event Orchestration and route events through that, the timestamp and all other PD-CEF values turn up in the message:

{
  "client": "ITRS Geneos",
  "client_url": "https://www.itrsgroup.com",
  "contexts": [],
  "dedup_key": "_",
  "description": "Test Triggered",
  "details": {
    "value": "eta12345",
    "variable": "filename"
  },
  "event_action": "trigger",
  "payload": {
    "class": "sampler1",
    "custom_details": {
      "value": "eta12345",
      "variable": "filename"
    },
    "group": "FTSE",
    "severity": "critical",
    "source": "mine",
    "summary": "Test Triggered",
    "timestamp": "2022-10-31T10:13:54Z"
  },
  "routing_key": "XXX"
}

Edit: The timestamp is still ignored, however.